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Final  Report 


OVER  ARCHING  GOAL 

The  overarching  goal  was  to  prepare  geostationary  satellite  atmospheric  motion  vectors 
(AMV)  for  numerical  model  assimilation  experiments  and  test  for  improvements  in  the 
NAVY  global  numerical  model  (NOGAPS)  forecasts  of  tropical  cyclones  (TC)  using 
targeting  information  provided  by  NRL-MRY.  This  was  the  primary  motivation  for  this 
investigation. 

PRIMARY  OBJECTIVE 

Develop  an  AMV  processing  and  data  assimilation  strategy  to  optimize  positive  impacts 
on  NOGAPS  forecasts  of  TC  tracks  and  intensity. 

RESULTS:  EXECUTIVE  SUMMARY 

Experiments  were  performed  testing  the  impact  of  modifying  the  assumed 
observation  error  for  assimilating  AMVs  in  the  NRL  Atmospheric  Variational  Data 
Assimilation  System  (NAVDAS).  A  first  experiment  changed  the  observation  error  for 
all  Geostationary  AMVs  to  be  1.5  times  the  model  background  error.  This  weighting  is 
based  on  comparisons  with  an  adjoint-based  impact  parameter.  A  second  experiment  kept 
the  operational  observation  error  settings  for  AMVs,  but  used  experimental  “expected 
error”  quality  indices  on  CIMSS-produced  AMVS  to  reject  AMVs  that  were  indicated  to 
have  poor  quality.  A  third  experiment  set  the  observation  error  of  the  CIMSS-produced 
AMVs  equal  to  the  expected  error  indices. 

All  three  experiments  produce  forecast  differences  that  are  overall  slightly  positive  in 
terms  of  forecast  skill  but  with  low  statistical  significance.  The  observation  error  ratio 
experiment  produced  Northern  and  Southern  hemisphere  extra-tropical  500-hPa  height 
anomaly  correlations  that  are  slightly  better  than  the  control.  The  tropical  500-hPa  height 
anomaly  correlations  are  slightly  worse.  For  the  expected  error  threshold  experiment  the 
forecast  impact  in  the  southern  hemisphere  is  slightly  better  than  the  control  while  the 
northern  hemisphere  and  tropical  forecasts  are  slightly  worse.  The  observation 
error/expected  error  experiment  produced  very  little  difference  from  the  control. 
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1.  Introduction 


Geostationary  satellite-derived  Atmospheric  Motion  Vectors  (AMVs)  are  used  by 
numerical  weather  prediction  (NWP)  centers  around  the  world.  Although  their  forecast 
impacts  are  generally  beneficial,  the  complexity  of  AMVs  and  their  associated 
observation  errors  make  them  challenging  to  assimilate.  The  AMVs  operationally 
assimilated  in  NAVDAS  (the  NRL  Atmospheric  Variational  Data  Assimilation  System) 
are  produced  by  the  University  of  Wisconsin’s  Cooperative  Institute  for  Meteorological 
Satellite  Studies  (CIMSS),  the  National  Environmental  Satellite  Data  and  Information 
Service  (NESDIS),  the  U.S.  Air  Force  Weather  Agency  (AFWA),  the  European 
Meteorological  Satellite  Organization  (EUMETSAT),  and  the  Japan  Meteorological 
Agency  (JMA).  AMVs  valid  within  a  six-hour  time  window  around  the  NAVDAS 
analysis  time  are  ingested  for  assimilation.  To  reduce  the  data  volume,  these  AMVs  are 
superobbed.  AMVs  from  different  sensors,  satellites,  and  processing  centers  are  all 
superobbed  separately.  Details  of  the  superobbing  scheme  are  presented  in  Pauley  (2003). 

AMVs  have  a  large  and  beneficial  impact  on  NOGAPS  (Navy  Operational  Global 
Atmospheric  Prediction  System)  forecast  skill.  For  example,  as  described  in  Langland 
and  Baker  (2004),  AMVs  produce  the  second-largest  reduction  in  NOGAPS  24-hr 
forecast  error,  following  rawinsonde  observations  in  the  Northern  Hemisphere  and 
Advanced  Microwave  Sounding  Unit  (AMSU-A)  observations  in  the  Southern 
Hemisphere.  Although,  on  average,  their  impact  is  beneficial  (e.g.,  their  assimilation 
reduces  model  forecast  error),  the  impact  of  individual  AMVs  (and  other  types  of 
observations)  has  a  statistical  distribution  that  includes  both  beneficial  and  non-beneficial 
forecast  impacts.  This  distribution  of  impacts  is  due  to  our  inability  to  exactly  specify  the 
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true  errors  of  the  observations  and  the  background  (model  first-guess),  which  are 
combined  in  the  data  assimilation  process  to  produce  the  analysis  or  initial  conditions.  If 
these  errors  can  be  more  precisely  tuned,  the  observations  will  have  a  greater  beneficial 
impact  on  model  forecast  skill. 

This  report  will  describe  three  NOGAPS  model  forecast  experiments  in  which  we 
adjust  the  quality  control  and  specified  observation  errors  for  AMV  assimilation  into 
NAVDAS.  In  Section  2  we  use  information  from  the  observation  impact  procedure 
developed  by  Langland  and  Baker  (2004).  In  Section  3,  we  use  a  newly  available  AMV 
quality  indicator,  referred  to  as  the  “expected  error”,  and  conduct  two  forecast  impact 
experiments.  The  results  of  these  three  experiments  are  then  summarized  and  followed  by 
discussion  and  ideas  for  future  efforts. 


2.  Adjoint-based  Observation  Impact  Results 

Langland  and  Baker  (2004,  LB04  hereafter)  describe  a  procedure  to  estimate  the 
impact  of  individual  observations  on  a  measure  of  short-range  forecast  error.  This  method 
is  useful  for  determining  which  observations  improve  the  forecast,  and  regional  patterns 
in  observation  impacts  on  the  forecast.  The  LB04  observation  impact  equation  is: 


Se *  = 


(i y-Hxb),KT 


dJf 


(1) 


where  y-Hxh  is  the  innovation  (difference  between  the  observation  and  the 


background,  -  and  -  are,  respectively,  the  sensitivity  of  forecast  errors  on  the 

OX  a  dXh 

analysis  and  background  model  trajectories,  and  K1  is  the  transpose  of  the  Kalman  gain 
matrix  represented  by  the  data  assimilation  procedure.  The  quantity  “«/”  is  an  energy- 
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weighted  forecast  error  norm  that  includes  temperatures,  winds,  moisture,  and  surface 
pressure  from  the  surface  to  about  300  hPa.  In  these  results,  /  represents  a  24-hour 
forecast.  The  (  )  brackets  represent  a  scalar  inner  product.  The  observation  impact  SeKf  is 

an  approximation  of  the  increase  or  decrease  in  forecast  error  associated  with  the 
assimilation  of  observations.  In  this  context,  negative  values  of  Sesf  imply  a  forecast  error 

reduction,  which  is  a  “beneficial”  impact.  We  can  use  Sej  to  represent  the  impact  of  all 

observations,  or  any  arbitrary  subset  of  observations.  This  adjoint-based  method 
provides  information  about  observation  impact  that  is  difficult  or  impossible  to  obtain 
from  conventional  data  denial  experiments. 

Using  the  LB04  method,  observation  impact  values  were  calculated  for  0000  UTC 
analyses  from  3-10  July  2007,  and  include  all  geostationary  AMV  data  used  in 
NAVDAS  operations.  Figure  1  and  Figure  2  (top  panels)  shows  the  observation  impact 
sum  binned  as  a  function  of  wind  component  (u  or  v)  speed  (Figure  1)  and  wind  direction 
(Figure  2).  The  lower  panels  of  each  figure  show  the  corresponding  number  of 
observations  in  each  bin.  These  figures  show  that  the  total  impact  is  fairly  well  correlated 
with  the  number  of  observations  in  each  bin.  Thus,  there  is  no  strong  bias  for  lower  or 
higher  wind  speeds,  or  winds  from  any  particular  direction  to  have  larger  impact  on 
short-range  forecast  skill. 
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Observation  Value  Bin  (ms"1) 


Figure  1:  Observed  AMV  wind  component  value  (m  s"1)  vs.  summed  observation  impact  (top 
panel)  and  number  of  observations  (lower  panel). 


Figure  2:  Observed  AMV  wind  direction  (degrees)  vs.  summed  observation  impact  (top  panel) 
and  number  observations  (lower  panel) 
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Impact  Sum  Binned  by  Latitude 


Figure  3:  Observed  AMV  latitude  (y-axis)  vs.  summed  observation  impact  (top  panel)  and 
number  of  observations  (lower  panel).  AMVs  have  the  largest  impact  just  south  of  the  equator, 
although  there  are  more  observations  just  north  of  the  equator.  This  may  be  explained  by  the 
larger  number  of  other  upper-air  observations  (e.g.,  radiosondes,  aircraft  observations)  in  the 
Northern  Hemisphere  that  reduces  somewhat  the  impact  of  AMVs. 
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In  figures  3-5  we  show  AMV  observation  impact  and  observation  counts  as 
functions  of,  respectively,  observation  latitude,  longitude,  and  pressure.  Figure  3 
illustrates  that  the  maximum  observation  impact  is  near  10°  S,  with  a  secondary  peak 
near  10°  N  latitude.  Although  the  largest  number  of  AMV  observations  is  near  10°  N, 
there  are  also  more  competing  types  of  upper  air  observations  (.e.g,  radiosondes,  aircraft) 
north  of  the  equator,  which  reduces  the  total  impact  of  the  AMVs.  Thus,  the  peak  impact 
of  AMV  observations  occurs  just  south  of  the  equator,  where  AMVs  contribute 
proportionately  more  to  the  analysis. 

As  a  function  of  longitude  (Figure  4),  the  observation  impact  is  generally  proportional 
to  the  number  of  observations,  with  a  peak  in  the  Atlantic  basin,  near  45°  W.  The  large 
number  of  AMV  observations  here  is  probably  caused  by  an  overlap  in  coverage 
provided  by  GOES-E  and  Meteosat-9. 

The  plot  of  AMV  impact  as  a  function  of  pressure  (Figure  5)  shows  several  important 
features.  The  AMVs  impact  the  forecast  in  two  regions:  in  the  lower  troposphere  near 
850-hPa  and  near  the  level  of  the  upper  tropospheric  jet  streams  near  400-hPa.  The  850- 
hPa  maximum  is  slightly  above  the  level  (900  hPa)  where  the  number  of  lower 
tropospheric  AMVs  is  largest,  and  the  400-hPa  slightly  below  the  level  (300-hPa)  where 
the  number  of  upper  tropospheric  AMVs  is  largest.  As  AMV  observation  error  is  soley  a 
function  of  pressure,  this  result  suggests  that  the  specification  of  observation  and 
background  errors  in  NAVDAS  is  not  optimal  at  certain  levels.  Specifically,  it  appears 
that  AMVs  at  300-hPa  and  900-hPa  should  have  larger  beneficial  impact  than  is  shown  in 
Figure  6. 
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Impact  Sum  Binned  by  longitude 


Number  of  obs  binned  by  Longitude 


Figure  4:  Observed  AMV  longitude  vs.  summed  observation  impact  (top  panel)  and  number  of 
observations  (lower  panel). 


Impact  Sum  Binned  by  Pressure 


Figure  5:  Observed  AMV  pressure  vs.  summed  observation  impact  (top  panel)  and  number  of 
observations  (lower  panel).  Peak  impacts  occur  at  about  850  hPa  and  400  hPa. 
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Data  assimilation  procedures  create  analyses  of  atmospheric  fields  by  blending 
information  from  the  background  with  new  observations.  An  observation’s  influence  on 
the  analysis  is  determined  largely  by  the  ratio  of  the  assumed  error  of  that  observation 
with  the  assumed  error  of  the  background  (a  short-range  model  forecasts)  at  the 
observation  location.  Observations  that  are  believed  to  be  more  accurate  (e.g.,  smaller 
observation  error)  receive  more  weight  than  do  less  accurate  observations.  In  NAVDAS, 
the  assumed  error  of  wind  observations  is  a  function  of  pressure  level  (see  Figure  7).  The 
true  observation  and  background  errors  are  unknown,  so  a  large  fraction  of  observations 
consequently  produce  either  too  much  or  too  little  influence  on  the  analysis. 

It  is  useful  to  examine  the  correlation  of  observation  impact  with  the  assumed 
observation  and  background  errors.  Figure  6  shows  the  summed  AMV  observation 
impact  plotted  with  the  ratio  of  the  assumed  observation  error  <rob  and  the  assumed 

background  error  <rhk.  This  ratio  is  The  AMV  observation  impact  is  most 

CTbk 

beneficial  when  Xob  =1.5,  a  ratio  of  AMV  observation  error  and  NOGAPS  background 
error  that  is  close  to  “optimal”  for  the  current  NAVDAS.  Why  this  is  so  needs  to  be 
explored  further.  Slightly  more  observations  have  =1.4,  but  these  observations  have 

less  beneficial  impact,  even  though  they  are  assumed  to  be  more  accurate  relative  to  the 
background.  This  suggests  that  the  observation  error  for  these  AM  Vs  may  be 
underestimated  or  over-estimated.  There  are  also  larger  numbers  of  observations  for 
which  Xob  >  1.6,  but  again  their  summed  impact  is  again  less  beneficial  than  those  for 

Xob  =  1 .5.  As  observation  error  increases,  they  receive  less  weight  in  the  data  assimilation 
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procedure,  and  their  impacts  should  become  smaller.  Note  that  <roh  applies  to  the 
superobbed  AMY  observations  not  the  individual  AMVs. 


Figure  6:  Ratio  of  assumed  AMV  observation  error  /  background  error  vs.  summed  observation 
impact  (top  panel)  and  number  of  observations  (lower  panel).  Observations  with  the  most 
beneficial  impact  have  observation  errors  that  are  1 .5  times  as  the  value  of  the  background  error. 


3.  Observation  and  Background  Error  Ratio  Experiment  (Exp  1) 

In  this  section  we  describe  an  experiment  to  test  the  hypothesis  that  NOGAPS 
forecasts  will  be  improved  if  AMV  observation  errors  are  adjusted  to  impose  a  ratio  of 
Kb  ~  1-5.  The  current  operational  AMV  observation  errors  are  shown  as  red  circles  in 
Figure  7  as  a  function  of  pressure.  The  minimum  observation  error  is  2.8  ms'1  at  the  low 
levels  and  increases  to  a  maximum  of  5.2  ms'1  above  300  hPa.  The  green  circles  represent 
the  observation  errors  that  result  when  Aob  =  1.5,  retaining  the  operational  values  of  the 
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assumed  background  wind  error.  At  most  pressure  levels,  this  change  implies  small 
reductions  (<  1  ms’1)  of  assumed  AMV  observation  error.  However,  there  are  small 
increases  in  observation  error  near  400  hPa  and  from  about  800-950  hPa. 


New  and  Old  Observation  Errors  vs.  Pressure 


Figure  7:  AMV  observation  errors  as  a  function  of  pressure.  The  red  circles  are  the  operational 
NRL  AMV  observation  errors.  The  green  observation  errors  are  the  observation  errors  that  result 
if  the  observation  error  equals  1 .5  times  the  background  error. 

In  this  experiment  two  data  assimilation  and  forecast  cycles  are  run,  daily,  from  1 
April  to  30  April  2007.  We  use  all  NOGAPS  operational  observations,  except  for  the 
substitution  of  special  CIMSS  AM  Vs  that  include  an  “expected  error  quality  indicator” 
(see  Section  4).  The  control  forecast  uses  the  operational  AMV  observation  errors.  The 
experimental  forecast  uses  the  exact  same  set  of  observations  as  the  control  run,  but 
changes  the  assumed  AMV  observation  error  so  that  Kb  1*5-  This  change  is  imposed 
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for  all  AM  Vs  in  the  global  domain,  at  all  pressure  levels.  The  forecast  impact  results  of 
this  experiment  are  described  in  Section  7. 

4.  AMV  Quality  Indicators  and  Expected  Error  Estimates 

As  described  above,  specifying  the  appropriate  observation  error  is  critical  to  data 
assimilation  and  forecast  quality.  Specifying  it  correctly  for  AMVs,  however,  is  relatively 
difficult.  The  derived  nature  of  the  product  using  multiple  satellite  images,  along  with 
assumptions  about  background  winds  provided  by  forecast  models  makes  estimating  the 
true  error  of  these  observations  difficult.  It  has  also  been  shown  that  AMV  observation 
errors  are  spatially  correlated,  and  this  complexity  is  not  handled  well  by  current 
variational  data  assimilation  procedures. 

Traditionally,  AMV  producers  have  not  estimated  the  observation  error  directly; 
Instead,  they  assign  normalized  quality  indicators  to  assess  whether  a  particular  AMV  is 
“good”  or  not.  Operational  AMV  producers  generate  two  objective  methods:  the 
recursive  filter  (RF)  method  (Velden  et  al  1998)  and  the  automatic  quality  indicator  (QI) 
method  (Holmlund  1998).  Both  techniques  assign  a  normalized  indicator  that  measures 
the  vector’s  fit  to  its  neighbors,  consistency,  and  fit  to  a  background  model  (Velden  et  al 
2005). 

Recently,  CIMSS  has  been  experimenting  with  a  quality  indicator  derived  by 
LeMarshall  et  al  (2004).  This  indicator,  called  the  “expected  error”  (EE),  linearly 
regresses  the  predictors  that  make  up  the  QI  along  with  additional  vector  and  model 
information  against  co-located  AMV  -  rawinsonde  differences.  A  list  of  the  regression 
components  and  values  for  GOES- 12  AMVs  for  various  channels  is  provided  in  Table  1 . 
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Predictor 

IR 

Visible 

Water- Vapor 

Shortwave  IR 

Constant 

10.1 

2.92 

9.33 

8.85 

QI  Speed  Test 

-0.180 

-2.02 

-0.366 

1.86 

QI  Direction  Test 

-0.36 

-1.83 

-0.224 

1.72 

QI  Vector  Test 

-0.99 

1.88 

-0.739 

-2.66 

QI  Buddy  Test 

-1.83 

-1.72 

-1.72 

-2.02 

QI  Forecast  Test 

-3.02 

-1.90 

-3.56 

-1.45 

A  MV  Speed 

0.092 

0.183 

0.101 

0.06 

AMV  Pressure 

-3.14xl0'3 

3.01xl0'3 

-9.28x1  O'4 

-5.5x1  O'3 

Model  Wind  Shear 

0.405 

0.012 

0.022 

0.091 

Model  Temp.  Gradient 

0.030 

-0.025 

0.0290 

0.031 

Table  1:  Expected  Error  linear  regression  coefficients  for  Northern  Hemisphere  GOES- 12 
Infrared,  visible,  water  vapor,  and  shortwave  IR  AMVs. 

Figure  8  and  Figure  9  show  two  measures  of  the  EE’s  skill  and  highlight  the  two  ways 
it  will  be  used  in  this  study.  Infrared  AMVs  from  GOES- 12  are  co-located  with 
rawinsondes.  The  results  shown  below  were  generated  from  AMVs  collocated  with 
rawinsondes  from  September  -  November  2007.  The  EE’s  performance  varies  little  from 
month  to  month.  Box-and-whisker  plots  of  AMV-rawinsonde  distributions  grouped  by  a 
central  expected  error  bin  value  (i.e.  a  bin  of  6  ms'1  contains  AMVs  that  have  expected 
errors  between  5.5  ms'1  and  6.4  ms'1)  are  shown  in  Figure  8.  The  1-1  line  is  shown  for 
comparison.  If  the  EE  were  a  perfect  predictor  all  of  the  AMVs  would  fall  on  this  line. 
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Although  many  of  the  AMVs  do  not,  most  of  the  mean  and  median  values  of  the 
distributions  do  fall  fairly  close.  The  mean  and  medians  of  the  AMV  -  rawinsonde 
difference  distributions  generally  increase  with  increasing  EE.  Analyzing  the  EE  in  this 
way  suggests  that  it  has  skill  at  estimating  AMV  error  compared  to  rawinsonde  error. 
Even  if  the  EE  were  a  perfect  correlative  measure  of  AMV  -  rawinsonde  differences, 
however,  it  would  only  be  an  approximation  of  the  true  AMV  observation  error.  The  EE 
does  not  take  into  account  rawinsonde  measurement  or  representative-ness  error  between 
rawinsondes  and  AMVs,  nor  does  it  account  for  errors  introduced  by  imprecise 
collocations. 


NHIR  Expected  Error  Distribution 


Figure  8:  Box  and  whisker  plots  showing  the  distributions  of  actual  AMV  -RAOB  (rawinsonde) 
differences  as  binned  by  a  central  expected  error.  The  lower  line  of  the  box,  the  red  line  in  the 
middle  of  the  box,  and  the  top  line  of  the  box  represent  the  lower  quartile,  the  median,  and  the 
upper-quartile  of  the  distributions,  respectively.  The  red  crosses  are  outliers.  The  average  (green 
circle)  and  rms  (red  square)  values  for  each  distribution  are  also  shown.  A  1-1  dotted  line  is 
shown  for  comparison. 
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As  discussed  in  Section  3,  estimating  the  observation  error  is  important.  But  the  EE 
can  also  be  used  to  decide  whether  a  given  observation  should  be  assimilated  or  not. 
Figure  9  bins  the  AMV  -  rawinsonde  differences  by  the  maximum  EE  rather  than  a 
central  value  as  in  Figure  8.  The  1-1  line  is  also  shown.  This  figure  shows  that  errors 
decrease  as  the  maximum  threshold  decreases.  On  average,  however,  they  do  not 
decrease  by  an  appreciable  amount  until  the  EE  thresholds  drop  below  about  7  ms'1.  Most 
of  the  EEs  are  below  this  value  (see  Figure  10).  As  the  threshold  gets  lower,  the  error 
distributions  move  toward  lower  errors  as  well,  reflecting  the  EEs  skill  at  thresholding 
out  bad  AM  Vs. 


Figure  9:  As  in  Figure  8,  except  the  box  plots  represent  the  AMVs  that  have  the  maximum 
expected  error  value  shown  in  the  x-axis.  As  the  expected  error  maximum  threshold  decreases, 
the  errors  generally  decrease,  particularly  when  the  threshold  value  decreases  below  about  7  ms'1. 
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Expected  Error  threshold. 

Figure  10:  (Top  plot)  Expected  Error  histograms  (x  104)  for  GOES-E,  GOES-W  and  Metosat-9 
AMVs  for  April  1-2  and  April  15-16,  2007.  (Bottom  plot)  Cumulative  distribution  for  the  same 
dataset. 

The  results  shown  in  Figure  8  and  Figure  9  are  confined  to  only  those  AMVs  that  are 
co-located  with  rawinsondes.  AMVs  not  necessarily  matched  with  rawinsondes  are 
shown  in  Figure  10:  a  histogram  and  cumulative  probability  distribution  of  the  EE  from 
four  days  of  AMVs  during  the  month  of  April,  2007.  The  average  EE  is  about  4  ms'1  and 
most  of  the  EEs  are  below  8  ms'1.  These  results  are  reasonably  consistent  with  the  results 
for  AMVs  co-located  with  rawinsondes. 
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5.  EE  Threshold  Experiment  (Exp  2) 

The  AMV  EE  information  can  be  used  to  design  additional  data  assimilation  and 
forecast  experiments.  In  Experiment  2  we  use  the  EE  to  define  AMV  quality  threshold 
values  and  withhold  poor-quality  AMVs  from  being  assimilated.  If  the  threshold  value  is 
set  too  tight,  then  too  many  AMVs  are  rejected  and  the  experiment  will  not  have  much 
impact.  If  the  threshold  is  set  too  loose,  then  too  few  AMVs  are  rejected,  and  the  analysis 
may  be  degraded.  Another  consideration  is  that  the  EE  is  a  function  of  AMV  speed. 
Faster  AMVs  should  therefore  be  given  a  higher  EE  tolerance.  Based  on  an  analysis  and 
tests  with  NRL’s  superob  procedure,  the  threshold  values  in  Table  2  were  selected  For 
Experiment  2. 


AMV  Speed 

EE  Threshold 

Speed  <  50  ms'1 

EEs  >7.5  ms’*  rejected 

Speed  >  50  ms'1 

EEs  >15  ms'1  rejected 

Table  2:  EE  Thresholds  used  in  Experiment  2 


In  this  experiment,  the  control  forecasts  use  operational  quality  control  procedures 
and  observation  error  values.  The  EE  threshold  forecasts  use  the  values  in  Table  2  to 
remove  selected  AMVs  before  the  assimilation  procedure.  The  forecast  impact  results  of 
this  experiment  are  described  in  Section  7. 

These  EE  thresholds  were  applied  to  a  special  set  of  CIMSS  AMVs  from  the  period 
of  1  -  3 1  April  2007,  which  include  the  expected  error  information.  Note  that  the  CIMSS 
winds  are  only  a  subset  of  all  AMVs  in  the  global  domain.  How  much  of  a  subset?  An 
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example  of  typical  AMV  numbers  is  shown  in  Table  3.  The  table  shows  a  sample  of 
AMV  superobs  generated  from  AMVs  from  each  regional  data  provider  and  compares 
them  to  superobs  generated  from  AMVs  produced  at  CIMSS.  Although  the  counts  are 
from  July,  2007,  these  numbers  are  not  likely  to  vary  appreciably  from  month  to  month. 
With  the  exception  of  Meteosat-9,  CIMSS  produces  fewer  AMVs  than  the  other 
producers.  This  difference  may  be  to  due  to  quality  control  differences  between  the 
producers.  As  only  the  CIMSS  AMVs  are  affected  by  using  the  expected  error,  the 
numbers  in  the  CIMSS  column  represent  the  number  of  observations  that  will  be  affected 
by  the  thresholding  and  the  subsequently  described  expected  error-observation  error 
experiment.  Keep  in  mind,  however,  that  the  geographic  distribution  between  CIMSS  and 
the  other  producers  is  nearly  identical.  Thus,  modifying  only  the  CIMSS  AMVs  should 
not  have  a  geographic  bias. 


NESDIS 

CIMSS 

GOES 

350850 

313884 

EUMETSAT 

CIMSS 

Met-7 

120414 

76754 

EUMETSAT 

CIMSS 

Met-9 

148154 

355134 

JMA 

CIMSS 

MTSAT 

119546 

102068 

Others 

CIMSS 

Total 

738964 

847840 

Table  3:  Number  of  AMV  superobs  (all  channels)  from  July  2007  generated  from  AMVs  by  each 
satellite  from  CIMSS  and  the  respective  national  producer.  Although  CIMSS  AMVs  represent  the 
majority  of  total  AMVs,  the  modifications  in  experiment  2  and  3  are  not  made  to  all  of  the 
available  AMVs. 
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6.  EE  and  Observation  Error  Experiment  (Exp  3) 

In  Experiment  3,  all  of  the  observations  are  assimilated  as  in  the  control,  but  the  EE 
replaces  the  operational  observation  error  for  all  CIMSS  AMVs.  Figure  1 1  is  a  scatter 
plot  of  operational  AMV  observation  error  versus  the  EE.  For  most  AMVs  the  EE  is 
somewhat  larger  than  the  operational  error,  so  that  the  observation  will  receive  somewhat 
less  weight  in  the  analysis.  This  is  particularly  true  for  AMVs  with  higher  speeds  that 
generally  have  higher  EE.  Some  AMVs,  however,  have  particularly  low  EE.  These  low 
EE  AMVs  are  generally  slow  in  speed,  and  in  the  low  levels. 


Figure  11:  Scatter  plot  of  operational  AMV  observation  error  and  “expected  error”  for  AMVs 
used  in  Experiment  3.  Color  scale  corresponds  to  AMV  speed. 
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Experiment 

Description 

1 .  Observation  and  Background  Error 

Ratio  Experiment 

Observation  errors  for  all  AM  Vs  are  modified  so  that 

the  observation/background  error  ratio  is  1 .5 

2.  Expected  Error  Quality  Threshold 

Experiment 

CIMSS-produced  AMVs  are  rejected  if  their 

expected  errors  are  above  set  thresholds 

3.  Expected  Error  as  Observation 

Error  Experiment 

Observation  errors  for  CIMSS-produced  AMVs  are 

set  to  expected  error. 

Table  4:  Summary  of  data  impact  experiments. 


7.  Experiment  Results 

This  section  describes  the  forecast  impact  results  of  Experiments  1 ,  2,  and  3  that  are 
configured  as  in  Table  4. 


7.1  Experiment  1-  AMV  Observation  and  Background  Error  Ratio 

Modifying  the  AMV  observation  errors  so  Xob  =  1.5  produced  neutral  to  slightly 
positive  impact.  Figure  12  shows  the  500-hPa  anomaly  correlations  for  the  Northern 
Hemisphere,  Southern  Hemisphere,  and  the  Tropics.  None  of  the  anomaly  correlation 
differences  are  statistically  significant.  On  average,  the  Northern  Hemisphere  and 
Southern  Hemisphere  experiments  were  slightly  positive  (compared  to  the  control)  for 
long  forecast  times  while  the  tropical  result  was  slightly  negative  in  terms  of  500-hPa 
anomaly  correlation. 
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Figure  12:  Experiment  #1  anomaly  correlation  of  500  hPa  height.  Northern  Hemisphere  (top), 
tropics  (middle)  and  SH  (lower).  Control  forecasts  (red)  and  forecasts  with  modified  observation 
error  (green).  The  anomaly  correlation  differences  are  not  statistically  significant,  although 
forecast  skill  is  slightly  improved  outside  the  tropics  with  the  modified  observation  error. 
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Level 

24-hour 

48-hour 

72-hour 

96-hour 

120-hour 

850  hPa 

0.015 

0.037 

0.046 

0.056 

0.052 

700  hPa 

-0.028 

-0.011 

0.007 

0.017 

0.008 

500  hPa 

-0.057 

-0.053 

-0.042 

-0.021 

-0.04 

250  hPa 

-0.108 

-0.147 

-0.132 

-0.117 

-0.045 

Table  5:  Differences  of  root  mean  square  (RMS)  wind  vector  forecast  error  between  the  control 
and  modified  observation  error  forecasts  in  Experiment  1 ,  for  the  tropics  (20°S-20°N).  Positive 
numbers  indicate  that  forecasts  with  modified  observation  error  have  lower  forecast  error. 
Differences  in  bold  are  statistically  significant  with  at  least  90%  confidence. 

Experiment  1 ,  however,  did  produce  statistically  significant  results  in  terms  of  root 
mean  square  wind  forecast  error  differences  ( 

Table  5),  at  least  in  the  tropical  band  from  20°S  to  20°N.  In 
Table  5,  positive  numbers  indicate  that  the  RMS  forecast  error  of  the  modified 
observation  error  forecasts  is  lower  than  in  the  control  forecasts.  As  seen  in 

Table  5,  the  forecast  results  differ  substantially  by  level.  In  general,  forecast  errors 
were  reduced  below  700  hPa,  and  increased  to  some  extent  above  500  hPa  by  changing 
the  observation  error.  As  forecast  length  increases,  the  forecasts  with  modified 
observation  errors  become  somewhat  more  skillful  (or  less  unskillful),  although  these 
differences  are  not  statistically  significant.  Why  is  the  tropical  forecast  impact  different 
than  the  extra-tropical  forecast  impact  in  this  experiment?  We  might  hypothesize  that 
wind  observations  impact  the  tropics  more  than  the  extra-tropics.  Another  clue  may  be 
seen  in  Figure  13.  This  plot  shows  the  same  quantities  as  Figure  6,  except  for  the  Tropics 
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(20°N  -  20°S).  When  the  entire  global  domain  is  considered,  the  optimal  ratio  of 
observation  error  to  background  error  appears  to  be  about  1.5.  However,  for  the  tropics, 
the  ratio  appears  closer  to  1 .4,  suggesting  that  the  optimal  ratio  of  observation  error  to 
background  error  might  be  defined  as  a  function  of  latitude  as  well  as  pressure. 

Tropics  Impact  Sum  Binned  by  O-Error/B-  Error 


Figure  13:  As  in  Figure  6  except  for  the  tropics  (20°N  through  20°S).  The  maximum  forecast 
error  reduction  occurs  when  the  AMV  observation  error  is  1 .4  times  as  large  as  the  background 
error. 
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7.2  Experiments  2  and  3-  EE  Threshold  and  Observation  Error  Experiments 

Results  from  the  EE  in  Experiments  2  and  3  are  essentially  neutral  in  terms  of 
forecast  skill.  Figure  14  and  Figure  17  show  the  500-hPa  height  anomaly  correlations 
results  for  these  two  experiments,  in  which  the  control  forecasts  and  EE  forecasts  are  not 
statistically  different.  The  EE  threshold  experiment  (Figure  14)  shows  a  slightly  stronger 
forecast  impact  than  the  EE-observation  error  experiment  (Figure  1 7). 

At  forecast  times  longer  than  about  84hr,  the  EE-threshold  Southern  Hemisphere 
height  anomaly  correlation  is  slightly  better  at  500-hPa  and  1000-hPa  (not  shown).  The 
Northern  Hemisphere  control  forecasts  are  slightly  better  than  the  EE-threshold  forecasts. 

A  time  series  for  Northern  and  Southern  Hemisphere  120-hr  rms  forecast  errors  of 
250  hpa  heights  in  Experiment  2  is  shown  in  Figure  15.  The  EE-threshold  forecasts  have 
a  lower  mean  RMS  error  than  the  control  forecasts  in  the  Southern  Hemisphere  (0.670  m 
lower)  and  a  slightly  higher  mean  RMS  error  than  the  control  in  the  Northern  Hemisphere 
(0.636  m  higher).  Differences  in  the  Northern  Hemisphere  wind  field  (not  shown)  are 
also  worse  (but  not  significantly  worse)  than  the  control.  The  1 20-hr  tropical  wind  vector 
error  time  series  (Figure  16)  also  shows  the  control  forecast  being  slightly  better  than  the 
experiment  (0.073  ms'1).  These  differences  are  consistent  with  the  fact  that  changing  the 
AMV  assimilation  procedure  causes  greater  differences  in  the  Southern  Hemisphere, 
where  AM  Vs  comprise  a  relatively  larger  fraction  of  the  total  available  observations. 
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Figure  14:  Experiment  #2  anomaly  correlation  of  500  hPa  height.  Northern  Hemisphere  (top), 
tropics  (middle),  and  SH  (lower).  Control  forecasts  (red)  and  EE-threshold  forecasts  (green).  The 
anomaly  correlation  differences  are  not  statistically  different,  although  the  southern  hemisphere 
forecast  skill  is  slightly  improved  with  the  EE-threshold. 
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250  hPa  Height  120hr  RMS  Errors 


Figure  15:  Time  series  of  250  hPa  120-hr  RMS  height  error  for  the  control  forecasts  (red)  and 
EE-threshold  forecasts  (blue).  The  mean  error  of  the  EE-threshold  forecasts  is  slightly  lower  than 
the  mean  control  error  in  the  Southern  Hemisphere  and  slightly  higher  in  the  Northern 
Hemisphere,  although  these  differences  are  not  statistically  significant.  The  Northern  Hemisphere 
vector  RMS  (not  shown)  is  consistent  with  the  height  error. 
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Tropical  250  b Ea  Vector  120  hr  RMS  Errors 


Figure  16:  Time  series  of  250  hPa  120-hr  RMS  height  errors  for  the  control  forecasts  (red)  and 
EE-threshold  forecasts  (blue).  The  EE-threshold  forecasts  are,  on  average,  slightly  worse  than  the 
control  forecasts,  although  the  difference  is  not  statistically  significant. 
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Figure  17:  Time  series  of  250  hPa  120-hr  RMS  height  errors  for  the  control  forecasts  (red)  and 
EE-threshold  forecasts  (blue).  The  EE-threshold  forecasts  are,  on  average,  slightly  worse  than  the 
control  forecasts,  although  the  difference  is  not  statistically  significant. 
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Figure  18:  Experiment  #3  anomaly  correlation  of  500-hPa  height.  Northern  Hemisphere  (top), 
tropics  (middle)  and  SH  (lower).  Control  forecasts  (red)  and  EE-observation  error  forecasts 
(green).  The  anomaly  correlation  differences  are  not  statistically  different. 
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In  Experiment  #3,  differences  between  the  EE-observation  error  forecasts  and  the 
control  forecasts  are  relatively  small.  Figure  19  shows  the  250-hPa  wind  vector  RMS 
forecasts  error  for  the  Tropics.  The  control  and  EE-observation  error  forecast  differ  very 
little.  Similar  results  are  found  at  other  pressure  levels  and  forecast  times. 


Tropical  250  hPa  Vector  120  hr  RMS  Errors 


Figure  19:  Time  series  of  250  hPa  120-hr  RMS  height  errors  for  the  control  forecasts  (red)  and 
EE-observation  error  forecasts  (blue).  The  EE-observation  error  forecasts  are,  on  average,  slightly 
better  than  the  control  forecasts,  although  the  difference  is  not  statistically  significant. 

8.  Summary  and  Future  Work 

The  assimilation  of  atmospheric  motion  vectors  (AMVs)  provides  large  beneficial 
impact  to  NAVDAS  analyses  and  NOGAPS  forecast  skill.  At  present,  only  rawinsonde 
profiles  and  AMSU-A  radiance  observations  provide  larger  contributions  to  NOGAPS 
forecast  skill,  as  measured  by  a  global  forecast  error  norm  (Langland  and  Baker  2004). 
The  current  assimilation,  however,  of  AMVs  (and  other  observation  types)  should  not  be 
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considered  optimal.  A  major  challenge  is  to  provide  better  estimates  of  AMV  observation 
error,  since  this  affects  the  relative  weight  assigned  to  the  observation  versus  the 
background  in  the  data  assimilation  procedure.  Improper  specification  of  observation 
error  can  result  in  degradation  of  the  analysis  and  poorer  forecast  quality. 

Currently  in  NAVDAS,  the  AMV  observation  error  varies  only  by  observation 
pressure.  The  observation  error  is  not  a  function  of  latitude,  longitude,  or  observed  wind 
speed.  This  report  describes  three  experiments  that  test  modified  assumptions  for  AMV 
observation  error  and  quality  control.  All  three  experiments  produce  forecast  differences 
that  modestly  impact  the  forecast  skill,  but  with  low  statistical  significance. 

Result  1:  Observation  and  Background  Error  Ratio  Experiment 

The  AMV  observation  error  is  specified  at  1.5  times  the  background  error  for  all 
superobs  in  the  global  domain.  The  northern  and  southern  hemisphere  extra-tropical  500- 
hPa  height  anomaly  correlations  are  slightly  better  than  the  control.  The  tropical  500-hPa 
height  anomaly  correlations  are  slightly  worse. 

Result  2:  Expected  Error  Threshold  Experiment 

CIMSS-produced  AMVs  are  quality  controlled  using  the  EE  thresholds  described  in 
Table  2.  The  AMVs  produced  by  the  other  centers  are  left  unchanged.  Forecast  impact  in 
the  southern  hemisphere  is  slightly  better  than  the  control  while  the  northern  hemisphere 
and  tropical  forecasts  are  slightly  worse. 
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Result  3:  Expected  Error  Observation  Error  Experiment 

Observation  errors  for  the  CIMSS  AMVs  are  set  to  the  respective  EE  values.  The 
modified  and  control  forecasts  differ  very  little. 

These  results  provide  some  clues  and  suggest  that  additional  experimentation  is 
required  to  improve  the  current  assimilation  of  AMVs  in  the  NAVDAS.  Larger  forecast 
improvements  may  require  changes  to  AMV  super-ob  procedures  or  more  significant 
changes  to  AMV  observation  errors  that  affect  large  fractions  of  the  AMVs.  For 
example,  we  note  that  Experiment  1 ,  which  modified  the  observation  errors  of  all  of 
AMVs,  also  produced  the  larger  forecast  impact.  In  contrast,  the  two  EE  experiments 
affected  only  those  AMVs  produced  by  CIMSS,  and  thus  had  relatively  smaller  forecast 
impacts.  Future  experiments  might  use  EE  estimates  for  all  AMVs  in  the  global  domain. 

It  might  also  be  useful  to  introduce  a  latitude-dependence  in  the  AMV  observation 
error,  since  AMV  observation  representativeness  may  be  different  in  the  tropics  versus 
the  extra-tropics.  The  AMV  threshold  experiment  can  be  modified  to  make  the  threshold 
a  linear  function  of  speed  rather  than  using  simple  cut-off  values.  A  linear  threshold 
would  allow  additional  high-speed  AMVs  to  be  assimilated,  with  tighter  restrictions  for 
slower  AMVs.  The  results  in  Exp  1  clearly  indicate  some  correlation  with  pressure.  The 
1.5-modification  helped  the  low  level  AMV  assimilation  but  hurt  the  upper  levels.  This 
discrepancy  needs  further  investigation.  It  would  also  be  of  interest  to  calculate  the 
adjoint-based  observation  impact  as  a  function  of  the  AMV  EE.  Then,  one  could  better 
optimize  the  error  threshold  to  produce  the  most  beneficial  selection  of  AMVs  for  actual 
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assimilation.  The  goal  of  future  experiments  is  to  isolate  the  fraction  of  AM  Vs 
assimilated  in  NAVDAS  that  improve  the  analysis  and  forecast  quality. 
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